Method and system for interactive cosmetic enhancements interface

ABSTRACT

Provided is a method and system of providing a cosmetics enhancement interface. The method comprises showing, at the display screen of a computing device having a memory and a processor: a digital photograph including facial features; an interactive dialog portion reflecting a conversational input received and a subsequent response provided thereto from the computing device; and a product display portion; receiving an inquiry, as reflected in the interactive dialog portion, related to a cosmetic product for application onto a selected facial feature; receiving a selection of the cosmetic product based on a matching to the at least one facial feature according to a predefined rule; displaying, at the product display portion, a product representation associated with the selected cosmetic product; receiving an update request; and updating the digital photograph showing a modification to the facial feature on the display screen by simulating application of the selected cosmetic product thereon.

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/630,556 filed Dec. 15, 2011, and incorporates by reference thedisclosure of said Application No. 61/630,556 in the entirety.

FIELD

The present disclosure relates generally to a system and method forinteractive cosmetics enhancements interface, providing for a user tobrowse, virtually “try on”, find and purchase beauty products, amongother aspects.

BACKGROUND

Digitally-based detection of facial features based on an accesseddigital photograph is finding its way into a variety of internetapplications as well as modifications of the digital photograph usingvarious facial visual effects.

Trying on and buying beauty products can be a significant chore when itinvolves travelling to a store or boutique, then typically tryingseveral beauty products or articles before finding a satisfactory fit inaccordance with subjective tastes and prevailing trendiness. Personallytravelling to, and taking the time to try several products in the storeor boutique can result in a significant inconvenience to shoppers. Astreamlined process allowing shoppers/customers to save time, andfurther gain the benefit of expert opinions regarding recommendations ofbeauty products to personal preferences and other unique situationalcontext may advantageously provide time-saving benefits to the on-lineInternet shopper, enabling them to “try on”, in a virtual context viasimulation, any beauty article or product using a personal digitalphotograph, prior to making a purchase.

SUMMARY OF THE INVENTION

Provided is a method, executed in a computing device having a displayscreen and a processor, of providing a cosmetics enhancement interfacecomprising showing, at the display screen: a digital photographincluding a plurality of facial features; an interactive dialog portionreflecting a conversational input received and a subsequent responseprovided thereto from the computing device; and a product displayportion; receiving an inquiry, as reflected in the interactive dialogportion, related to a cosmetic product for application onto at least onefacial feature of the plurality of facial features; receiving aselection of the cosmetic product based on a matching to the at leastone facial feature according to a predefined rule; displaying, at theproduct display portion, a product representation associated with theselected cosmetic product; receiving an update request; and updating thedigital photograph showing a modification to the at least one facialfeature on the display screen by simulating application of the selectedcosmetic product thereon.

In an embodiment the conversational input comprises an audible voiceinput.

In another embodiment the conversational input comprises a text input.

In one variation of the method, the at least one facial feature consistsof one of a nose, at least one lip of a mouth, an eyebrow, an eyelid, afacial cheek, hair and an ear.

In another variation, the cosmetic product consists of one of lipstick,eye shadow, hair color, blush makeup, a cosmetic eye lash, and anarticle of jewellery.

In a further embodiment, the cosmetics enhancement interface comprises apurchasing tool portion to consummate an e-commerce transaction.

In yet another embodiment, the cosmetics enhancement interface furthercomprises a product advertisement section showing offers for purchase.

In one variation, the computing device consists of one of a desktopcomputer, a portable computer, a mobile wireless smartphone, and akiosk.

The computing device is communicatively coupled to a beauty productsdatabase in another variation.

Further to the method provided, the predefined rule for the matching isat least partly based on an event type specified in the conversationalinput.

In an embodiment, the computing device further includes a memory storinga plurality of earlier conversational inputs of a pre-identified user.The earlier conversational inputs may be used to create a beautypreferences profile of the pre-identified user.

In another embodiment, the product representation includes a pluralityof color options for selection of at least one option therefrom.

In, yet another variation, the product representation includes aprioritized list of product options for selection of at least oneproduct option therefrom. The product option selections may be used tocreate a beauty preferences profile of a pre-identified user.

In a further embodiment, the predefined rule for the matching is atleast partly based on the beauty preferences profile of thepre-identified user.

In yet another embodiment, the matching may be performed in theprocessor of the computing device.

Yet further, the cosmetics enhancement interface further may comprise avideo display portion.

In yet another embodiment, the cosmetics enhancement interface mayfurther comprise a plurality of hyperlinks to access websites associatedwith respective cosmetic products.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of example only, with referenceto the following drawings in which:

FIG. 1 shows an interactive interface according to an embodiment of theinvention;

FIG. 2 shows the interactive interface displaying an embodiment of theinteractive conversations with a user;

FIG. 3 shows the interactive interface displaying an embodiment of theinteractive conversations involving a situational context involving anevent specified by user;

FIG. 4 shows the interactive interface displaying an embodiment of theinteractive conversations involving a situational context involving apersonal article specified by a user for a style-based matching;

FIG. 5 shows the interactive interface displaying an embodiment of theinteractive conversations involving an instructional video related to aproduct recommendation; and

FIG. 6 shows the interactive interface displaying an embodiment of theinteractive conversations involving a purchasing tool for entering intoand consummating an ecommerce transaction related to a productrecommendation.

DETAILED DESCRIPTION

This description pertains to a system for assisting a beauty consumer(i.e. “user”) to browse, explore, virtually try-on, and/or purchasebeauty products, wherein a user starts by asking a question or requestrelated to her beauty interests or needs. Then, based on the user'sinitial question or request, the system will search the Internet and itsown internal database for relevant information about the product anddisplay the results to the user. The results could be in the form of abefore and after photo, a representative image, a video, a set of links,a scrolling list of entries, or any other mechanism for displayingvisual search results to the user. The user can then virtually try thebeauty products on her digital photo, or ask more questions via aninteractive conversation with the system. The user may then purchase theselected beauty products through an e-commerce website, mobile site, orapplication.

FIG. 1 shows an interactive interface according to an embodiment of theinvention.

A system is designed for assisting a user to browse, explore, virtuallytry on, and find the best beauty products. FIG. 1 shows an example ofthe user interface for the system. The large menu at the top of FIG. 1is the Virtual Makeover tool where the effect of applying differentmakeup products is simulated. The white box at the bottom of FIG. 1 isthe conversation tool provided to the user, which is intended to let theuser have a conversation to the system about her beauty needs.

The virtual makeover tool is a system that allows the user to visualizeany products, or effects on her own digital photo. This process is donecompletely automatically, meaning that all the interactions that userwill have with the virtual makeover system, is just to provide it with aphoto, and select different beauty products, or effects. As soon as theuser selects a product or effect, her photo will automatically beupdated accordingly.

The conversational tool is an system which allows the user to haveconversations to the system and get directions. The conversational toolcan work based on either voice or text input from the user. In eithercase, the input sentences from the user are detected and analyzed. Thenthe system uses the large data base of the information on the beautyproducts and their features, and provides the user with responses andguidelines.

A user starts by providing a digital photo of herself to the system, asshown in FIG. 1. This step could be done in different ways depending onthe system platform, which includes web, mobile, tablet, or kiosk. Thisstep is done because the system is then going to visualize the beautyproducts by means of automatically simulating the effect of applying themake up products on the user's face.

The user then proceeds by starting a conversation with the system,related to her beauty interests or needs. For example, the user canstart with the request:

I have blue eyes and I am looking for a beautiful eye shadow for myeyes.

Then, the system will respond back to the user with:

Okay. I can help you find an Eyeshadow. I just sorted the palette foryou.

This is shown in FIG. 1. What happens is that the virtual makeover toolprocesses the digital photo of the user, and finds the specificcharacteristics of the user's eyes. Then, based on those features, andthe color of the eyes mentioned by the user, the system automaticallydetects which eye shadows best match to the user's eyes. Finally thesystem sorts the palette of eye shadows for the user and provides herwith the option to select and try any of them. As soon as the userselects one of the suggested colors for the eyeshadow, an eyeshadow withthe same color will immediately be applied to her digital photo on thevirtual makeover tool, as shown in FIG. 1. This way, the user can try asmany colors as she wants and select the one she is interested in.

The system is designed in a way that it is not limited to request andquestions only. Instead, the user can also describe a situation to thesystem, and the system will guide them based on their situation. Anexample of this is shown in FIG. 2. In this example, the user hasexplained her situation to the system as:

I am going to a party tonight, and I was wondering what color is thebest for my lipstick.

The system has then reacted with the response:

For a party, I would recommend darker and more vibrant colors. I havesorted the lipstick products for you.

The lipstick products are then sorted accordingly, as shown in FIG. 2.In this case, the system does not only act as a simple automatedchatting tool, but it also has the ability to understand and analyze theexact beauty needs of the user and provide her with a professionalrecommendation via the interactive conversation tool, FIG. 2 furthershows the interactive interface displaying an embodiment of theinteractive conversations with a user.

FIG. 3 shows the interactive interface displaying an embodiment of theinteractive conversations involving a situational context involving anevent specified by user.

There are some situations that the system might be able to provide amuch better response to the user if it knows some more details about theuser's situation. In this case, the system asks the user for moreinformation about her situation. An example of this is shown if FIG. 3.The user has described her situation as:

I am going to a wedding. Could you help me choose a good blush?

The system then asks the following question in order to have moreinformation about the preferences of the user:

Sure, what is the color of your dress for the wedding?

The user's answer could be something like:

It is going to be purple.

Now the system knows the exact preferences of the user and can narrowdown the list of the beauty products for this specific user. So thesystem responds with:

Perfect, I have sorted the blush products that best match to your dresscolor for you.

And it sorts the colors of the blush products that best match the colorof the user's dress, as in FIG. 4, the interactive interface displayingan embodiment of the interactive conversations involving a situationalcontext involving a personal article specified by a user for astyle-based matching.

FIG. 5 shows the interactive interface displaying an embodiment of theinteractive conversations involving an instructional video related to aproduct recommendation. There is a component in the system which allowsbrowsing the web based on the user's question and presenting the resultsto the user. Therefore, all the information about the beauty products onthe web is used by the system in addition to the system's database forbeauty products. An example of this component is shown in FIG. 5. InFIG. 5, the user has asked the question:

How do I apply a smoky eye effect? Could you help me with that?

The system has then provided the response:

Sure, here is a video that might be helpful.

And it provides video of the specific brand for the user to watch.

FIG. 6 shows the interactive interface displaying an embodiment of theinteractive conversations involving a purchasing tool for entering intoand consummating an e-commerce transaction related to a productrecommendation.

The disclosure herein is not limited to the interactive conversationsprovided in the examples above, but it includes all kind of questionsand conversations about the features, pricing, usage, and all otherdetails of the beauty products.

There is a component in the interactive conversational system, whichkeeps track of the whole conversation, as well as the products alreadyselected by the user. So the details of the conversation and the make upproducts the user has tried and selected are all stored. Thisinformation is considered when providing responses to the user for thenext steps. Therefore, when the system is about to provide the responseto the user's question, it not only uses its own large database ofbeauty products and their features, but it also uses the history of theconversation it has had to the current user. So for example, if the userhas selected a green blush for her face, and now she is looking for alipstick, then the system can take this into account for recommending alipstick color to the user.

There is a component of the system that collects the overall results ofthe conversations with the users. The overall results include thedetails of the conversations, the details of the products tried andselected by the users. The system then utilizes these overall results,in order to update the large database of beauty products and theirfeatures that it uses to communicate to the users.

A component is also provided in the updating part of the system thatkeeps track of all the beauty products in the market. This componentthen updates the database of beauty products in the system according tothe products currently available in the market. This way, the system isalways synchronized with the most up to date beauty products.

The system collects all the beauty products selected by the user duringa conversation and displays them on the interface as shown in FIG. 6.The conversation will continue until the user is satisfied with thecollection of products that are visualized on her photo. At this, point,there is a component in the system that makes it possible for the userto purchase them through an e-commerce web or mobile site.

The disclosure herein is not limited to the makeup products on the faceonly. Rather, it pertains to other kinds of beauty products, such as butnot limited to hair color and dressing products. For all other kinds ofbeauty products, the user can have conversations to the system about thespecific product, and get directions for application or use.

Although the invention has been described with reference to specificexemplary embodiments in the disclosure herein, varying modificationsthereof will be apparent to those skilled in the art without departingfrom the scope of the invention as defined by the appended claims.

1. A computing platform having a processor and a memory, the memorystoring instructions which when executed by the processor providevirtual tools to simulate effects on a digital photo, the virtual toolsoperating to configure the computing platform to: generate a virtualdisplay of a digital photo including a face; automatically extractfacial features from the digital photo; determine a matched color andspecific cosmetic products to virtually try on the virtual display basedon an audio input or textual input related to the cosmetic products; andinteractively modify the virtual display including the face with one ormore of the specific cosmetic products.
 2. The computing platform ofclaim 1 wherein the audio input or textual input comprises aconversational input from a user.
 3. The computing platform of claim 2wherein to interactively modify comprises receiving selection input toselect among the specific cosmetic products to define the one or more ofthe specific cosmetic products and update the virtual display.
 4. Thecomputing platform of claim 3 wherein the virtual tools comprise aninteractive graphical user interface for displaying output and receivinginput comprising: a first portion in which to virtually display the facemodified with the one or more specific cosmetic products; a conversationportion to interactively receive the conversational input; and acosmetic products portion to display the specific cosmetic productsselected in response to the conversational input and to receive one ormore selection inputs to further select among the specific cosmeticproducts to define the one or more specific cosmetic products tovirtually try on.
 5. The computing platform of claim 4 wherein thevirtual tools operate to configure the computing platform to receive aplurality of selection inputs and responsive to each, interactivelymodify the virtual display with a respective one of the specificcosmetic products.
 6. The computing platform of claim 4 wherein thevirtual tools operate to configure the computing platform to: receiveadditional conversational input and determining a second matched colourand second cosmetic products to virtually try on the virtual display andinteractively modify the virtual display including the face with one ormore of the second specific cosmetic products.
 7. The computing platformof claim 2 wherein the virtual tools operate to configure the computingplatform to, in response to receiving a first portion of theconversational input: analyse the first portion of the conversationalinput; and generate and present a follow-up question to solicit anadditional portion of the conversational input.
 8. The computingplatform of claim 1 wherein the facial features comprise at least someof a nose, a lip, an eyebrow, an eyelid, a facial cheek and an ear. 9.The computing platform of claim 1 wherein the facial features comprise,a skin color, a skin texture, an eye color and a hair color.
 10. Thecomputing platform of claim 1 wherein the cosmetic products comprise alipstick, an eye shadow, a hair color, and a blush makeup.
 11. A methodexecuted in a computing platform having a processor and a memory, thememory storing virtual tools to simulate effects on a digital photo, themethod comprising: generating a virtual display of a digital photoincluding a face; automatically extracting facial features from thedigital photo; determining a matched color and specific cosmeticproducts to virtually try on the virtual display based on an audio inputor textual input related to the cosmetic products; and interactivelymodifying the virtual display including the face with one or more of thespecific cosmetic products.
 12. The method of claim 1 wherein the audioinput or textual input comprises a conversational input from a user. 13.The method of claim 2 wherein interactively modifying comprisesreceiving selection input to select among the specific cosmetic productsto define the one or more of the specific cosmetic products and updatethe virtual display.
 14. The method of claim 3 comprising providing aninteractive graphical user interface for displaying output and receivinginput comprising: a first portion in which to virtually display the facemodified with the one or more specific cosmetic products; a conversationportion to interactively receive the conversational input; and acosmetic products portion to display the specific cosmetic productsselected in response to the conversational input and to receiveselection input to further select among the specific cosmetic productsto define the one or more specific cosmetic products to virtually tryon.
 15. The method of claim 4 comprising receiving a plurality ofselection inputs and responsive to each, interactively modifying thevirtual display with a respective one of the specific cosmetic products.16. The method of claim 4 comprising receiving additional conversationalinput and determining a second matched colour and second cosmeticproducts to virtually try on the virtual display and interactivelymodifying the virtual display including the face with one or more of thesecond specific cosmetic products.
 17. The method of claim 2 comprising,in response to receiving a first portion of the conversational input:analysing the first portion of the conversational input; and generatingand presenting a follow-up question to solicit an additional portion ofthe conversational input.
 18. The method of claim 1 wherein the facialfeatures comprise at least some of a nose, a lip, an eyebrow, an eyelid,a facial cheek and an ear.
 19. The method of claim 1 wherein the facialfeatures comprise, a skin color, a skin texture, an eye color and a haircolor.
 20. The method of claim 1 wherein the cosmetic products comprisea lipstick, an eye shadow, a hair color, and a blush makeup.